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Abstract 

p  A  computational  model  was  used  to  improve  the  learnability  of  an  Air  Force  document, 
doubling  recall  and  greatly  improving  recruits'  mental  representation  of  the  content.  Kintsch's 
computer  model  of  reading  was  applied  to  a  1000  word  Air  Force  text  on  the  Air  Force's  role  in  Vietnam 
War.  Principles  of  the  model  were  used  to  identify  40  text  locations  where  recruits  would  have  to  make 
inferences  if  they  were  to  have  a  coherent  mental  representation  of  the  text.  Each  location  was  then 
repaired,  and  the  repaired  text  was  then  tested  for  learnability  against  the  original  text  in  two 
experiments.  In  Experiment  1,  free  recall  was  doubled  for  the  repaired  text.  In  the  second  experiment, 
120  recruits'  66-part  mental  representations  for  12  important  text  concepts  were  measured,  and 
compared  with  the  mental  representations  of  the  text's  author,  and  of  7  independent  subject  matter 
experts.  The  author  and  the  experts'  mental  representations  correlated  about  .80.  For  recruits 
who  read  the  repaired  text,  their  mental  representations  correlated  with  the  author  and  experts  about 
.55,  p  <  .05.  But  recruits  who  read  the  original  text  correlated  with  the  author  and  experts  only  about 
.10.  These  results  suggest  that  the  computational  model  can  be  used  to  improve  the  learnability  of  Air 
Force  texts.  Individual  differences  tests  of  inferencing  ability  were  developed. 
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The  objectives  of  this  project  were  to  use  a  computational  model  of  reading  to  find  out  how  to 
improve  learning  from  Air  Force  texts,  and  to  develop  an  individual  differences  model  based  on  the 
mental  processes  specified  in  the  model. 


Table  1 
Specific  Aims 

The  top  level  goal  is  to  produce  a  broadly  useful  way  of  improving  Air  Force  recruits'  learning 
from  instructional  text.  Since  about  half  of  Air  Force  training  is  based  on  text,  any  large  improvement  in 
learning  from  text  will  have  an  appreciable  impact  on  the  effectiveness  and  efficiency  of  Air  Force 
training.  The  project  is  based  on  Walter  Kintsch's  computer  model  of  text  comprehension.  The 
specific  aims  are  to: 

1 .  Analyze  two  Air  Force  texts  (one  on  the  history  of  the  air  war  in  Vietnam,  and  one  on  aerospace 
doctrine)  using  computer  programs  developed  by  Thea  Turner  and  Walter  Kintsch. 

2.  Use  the  theoretical  principles  underlying  the  programs  to  produce: 

(a)  a  tree  diagram  of  the  mental  representation  that  the  Kintsch  program  produces  after  reading 
the  text,  corresponding  to  the  mental  representation  produced  by  a  recruit  with  the  same 
abilities  as  are  embodied  in  the  Kintsch  program, 

(b)  a  tree  diagram  of  the  mental  representation  the  writer  intended  the  recruit  to  have  after 
reading  the  text,  corresponding  to  the  representation  produced  by  a  skilled  reader. 

3.  Add  abilities  to  the  Kintsch  program  to  produce  diagrams  of  the  mental  representations  of 
recruits  intermediate  between  highly  skilled  readers  and  those  only  as  skilled  as  the  Kintsch 
program.  (Most  of  these  additions  will  require  operator  assistance  to  run.)  The  creation  of  these 
additions  will  serve  as  a  heuristic  device  to  achieve  goals  4,  5,  and  6  below. 

4.  Develop  new  theoretically  motivated  individual  differences  tests  (these  will  measure  the  abilities 
added  to  the  program  in  goal  3).  Air  Force  recruits  in  the  Experimental  Testing  Facility  at 
Lackland  Air  Force  Base  will  be  used  to  develop  the  tests,  under  the  supervision  of  Patrick 
Kyllonen's  group. 

5.  Develop  a  model  relating  the  abilities  measured  by  the  individual  differences  tests  to  the  recruits' 
learning. 

6.  Revise  the  texts  so  that  they  do  not  require  the  individual  differences  abilities  specified  in  goal  3 
(or  require  less  of  the  abilities) 

7.  Perform  empirical  tests  of: 

(a)  the  individual  differences  tests  of  goal  4,  combined  as  in  the  model  of  goal  5,  for  their  efficacy 
in  predicting  recruits'  learning  of  the  original  Air  Force  texts, 

(b)  the  modified  versions  of  the  texts  (developed  in  goal  6)  for  their  efficacy  in  creating  the 
desired  representation  of  the  texts. 
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Current  Status  of  the  Research 


Accomplished  Objectives 

Using  Kintsch's  computational  model  of  reading  (Kintsch  &  Van  Dijk,  1978;  Miller  &  Kintsch, 
1980),  we  have  successfully  analyzed  and  rewritten  an  Air  Force  text  to  greatly  improve  its  learning: 
Free  recall  was  doubled,  and  the  shape  of  the  recruits'  mental  representation  of  the  rewritten  text  was 
much  closer  to  the  representation  of  the  texts  author  (and  of  7  independent  subject  matter  experts) 
than  the  recruits'  representation  of  the  original  text.  The  progress  of  this  work  is  described  in  narrative 
form  in  the  next  section. 

Narrative  Description  of  the  Accomplished  Objectives.  We  used  Kintsch's  model  to  analyze  a 
text  in  order  to  find  the  locations  in  the  text  at  which  the  model  had  a  problem  in  establishing 
coherence  while  “reading"  the  text.  Our  idea  was  that,  if  Kintsch's  model  was  a  good  simulation  of  what 
a  person  does  when  he  reads,  then  the  locations  in  the  text  where  Kintsch's  model  has  coherence 
problems  should  be  the  same  locations  where  the  person  will  have  problems. 

If  we  then  repaired  each  of  those  coherence  problems,  by  revising  the  text  to  make  coherence 
easy  to  establish,  then  the  model  would  no  longer  have  those  problems.  And  of  course  the  person 
who  is  reading  should  no  longer  have  those  problems  either,  if  the  model  is  a  good  model  of  the 
person.  The  result  should  be  that  a  person  reading  the  revised  text  should  understand  and  learn  it 
better. 


So,  our  first  step  was  to  analyze  the  text  to  find  out  the  locations  of  coherence  problems.  We 
used  the  procedure  shown  in  Table  2. 


Table  2 

Model-Based  Analysis  Procedure 

1.  Input:  “Naive-Literal"  Proposition  List. 

2.  When  coherence  fails  (no  argument  overlap):  the  program  stops. 

3.  We  count  an  inference  call  there. 

4.  Then  we  add  a  proposition  to  the  list  to  establish  coherence  (bridge  the  gap). 

5.  We  restart  the  program:  it  runs  (through  the  added  proposition)  until  either: 

(a)  coherence  fails  again  (go  to  3)  or 

(b)  the  program  finishes  the  passage 

6.  The  output  is  a  count  of  the  number  of  inference  calls. 

7.  Also  the  program  outputs  the  number  of  reinstatements  (another  problem  in  establishing 
coherence). 
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Study  1:  Correlations  Between  inference  Calls  and  Recall 

In  our  first  study  (Britton,  Van  Dusen,  Glynn  &  Hemphill,  1990)  we  tried  to  predict  text  recall 
with  those  numbers.  Specifically,  we  correlated  those  inference  counts  and  reinstatement  counts  with 
recall  and  short  answer  test  performance,  in  8  texts  revised  by  others.  The  texts  were  8  high  school 
history  texts  600-1000  words  long.  Each  text  was  recalled  by  24  undergraduates.  We  found 
correlations  around  -.8.  Table  3  shows  the  data  for  one  such  correlation. 


Table  3 

Example  of  a  Correlation  Between  Recall  and  Inference  Calls 


Passage/rewriters 

Immediate  recall  (%) 

Inference  Calls 

Vietnam/composition  instructors 

28 

3 

Korea/composition  instructors 

25 

3 

Vietnam/discourse  researchers 

23 

4 

Vietnam/original 

22 

4 

Korea/discourse  researchers 

20 

8 

Vietnam  /Time-Life 

20 

8 

Korea/original 

14 

8 

Korea/Time-Life 

13 

13 

Table  4  shows  the  entire  set  of  correlations  for  free  recall  and  short  answer  tests,  fo.  inference 
calls  and  for  reinstatements.  Table  3  shows  the  data  points  for  the  upper  left-most  correlation  in  Table 
4  (-.89). 


Table  4 

Inference  Cali  Correlations  with  Recall 


Free  Recall  Short  Answer 


Immediate 

Delayed 

Immediate 

Delayed 

Inference  calls 

-.89* 

-.85* 

-.78* 

i 

o> 

* 

Reinstatements 

-.80* 

-.75* 

-.71* 

i 

■~sj 

o 

4- 

*p  <  .05. 
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These  results  showed  we  could  predict  recall  and  short  answer  performance  quite  well  with 
numbers  that  the  Kintsch  model  calculates  from  the  text. 

Study  2:  Experimental  Demonstration  of  Effects  of  Repairing 
Inference  Calls  on  Mental  Representations 

Our  next  step  was  to  experimentally  demonstrate  that  the  model  could  be  used  to  control  the 
properties  of  a  text  to  make  it  more  learnable.  To  do  this,  we  chose  a  new  text  and  revised  it  using  the 
principles  of  the  Kintsch  program.  Table  5  shows  an  example  from  the  very  beginning  of  the  text, 
which  was  a  10 00  word  Air  Force  training  document  on  Air  Force  history  by  Major  Robert  Ehrhart,  a 
history  professor  at  the  U.  S.  Air  Force  Academy.  The  top  panel  shows  the  title  and  the  first  sentence 
from  the  Original  text. 


Table  5 

Example  of  Original  Text  and  Revision 


Original  Text 

Title:  Air  War  in  the  North,  1965 

By  the  fall  of  1964,  Americans  in  both  Saigon  and  Washington  had  begun  to  focus  on  Hanoi  as  the 
source  of  the  continuing  problem  in  the  South. 

1  ext  with  Inferences  in  Parentheses 

Title:  Air  War  in  North  (Vietnam)  1965 

By  the  fall  of  1964  (causing  events  in  1965),  Americans  in  both  Saigon  and  Washington  had  begun  to 
focus  on  Hanoi  (Capitol  of  North  Vietnam)  as  the  source  of  the  continuing  problem  in  the  South. 

Text  with  Inferences  Incorporated  into  Text 

Title:  Air  War  in  North  Vietnam,  1 965 

By  the  beginning  of  1965,  Americans  in  both  Saigon  and  Washington  had  begun  to  focus  on  Hanoi, 
Capitol  of  North  Vietnam,  as  the  source  of  the  continuing  problem  in  the  South. 


The  first  sentence  of  the  original  text  does  not  cohere  with  the  title,  according  to  the  Kintsch 
model,  since  there  is  no  argument  overlap,  that  is,  there  are  no  repeated  words.  To  establish 
coherence  we  added  to  the  proposition  list  (which  is  the  input  to  Kintsch's  program)  the  propositions 
shown  in  the  middle  panel  of  Table  5  (in  parentheses  in  Englished  form.)  And  also  we  rewrote  the  text 
to  incorporate  those  propositions;  the  result  of  the  rewriting  at  this  stage  looked  like  what  is  shown  in 
the  bottom  panel  of  Table  5. 
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Our  goal  in  this  revision  was  to  establish  coherence  wherever  the  author  intended  there  to  be 
coherence.  That  is,  we  thought  the  author  intended  people  to  connect  late  1964  with  1965,  and  to 
know  that  this  sentence  was  about  North  Vietnam,  the  same  North  Vietnam  referred  to  in  the  title.  We 
did  this  for  the  whole  text.  We  found  40  such  coherence  problems.  We  called  the  result  the  Principled 
Revision  because  we  analyzed  it  and  rewrote  it  according  to  the  principles  of  the  Kintsch  program. 

I  also  wrote,  in  advance,  a  Heuristic  Revision,  based  on  my  own  rules  of  thumb  for  writing  clear 
text.  I  expected  this  to  be  an  upper  control  for  the  Principled  Revision,  showing  what  could  be  done  if  I 
had  complete  freedom  to  rewrite  in  the  best  way  I  knew  how. 

Then  we  gave  each  version  to  22  undergraduate  students.  The  results  are  shown  in  Table  6. 


Table  6 

Results  of  Study  2 


Free  Recall 

Original 

Version 

Principled 

Revision 

Heuristic 

Revision 

(Number  of  Propositions) 

29 

63* 

57* 

Reading  Rate 
(words  per  minute) 

112 

128* 

112 

Length 

(words) 

1030 

1302 

1027 

Reading  Time 
(minutes) 

9.2 

10.2 

9.2 

Propositions 

Recalled  per  minute 

3.2 

6.1* 

6.2* 

of  Study  Time 


*p  <  .05,  different  from  original. 


The  recruits  recalled  twice  as  much  from  the  Principled  and  Heuristic  Revisions  as  from  the 
Original.  They  read  the  Principled  Revision  at  a  faster  rate.  Even  though  the  Principled  Revision  was 
longer,  people  didn't  take  significantly  longer  to  read  it.  And  they  recalled  more  propositions  per  unit 
time  from  it. 

Most  importantly,  we  also  compared  the  shape  of  people's  mental  representations  with  that  of 
the  author  and  other  experts.  To  do  this  we  picked  12  terms  from  the  text,  shown  in  Table  7.  We 
created  all  possible  pairs  of  these  terms,  which  is  66  pairs  in  all. 
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Table  7 

Some  Main  Terms  from  the  Texts 

Members  of  Johnson  Administration 

President  Johnson 


Civilian  Advisers  Military  Advisers 


Robert  McNamara  Maxwell  Taylor 


Military  Strategy 


Graduated  Response  Strategy 


Psychological 


Rolling  Thunder 


Success  Failure 


We  asked  separate  groups  of  40  Air  Force  Recruits  to  read  each  passage  and  then,  from 
memory,  to  rate  each  pair  of  terms  for  relatedness  on  a  7-point  scale:  66  pairs  in  all.  Then  we  asked 
the  author  -  Major  Ehrhart  -  to  read  his  original  text,  and  do  the  rating;  this  gave  us  the  author’s  mental 
representation.  Also  we  asked  seven  lor.ii  experts  (two  military  historians,  a  former  ambassador,  etc.) 
to  read  the  text  and  give  us  their  mental  representations.  The  average  correlations  between  the  author 
and  the  experts  was  .80. 

Then  we  correlated  the  66  items  for  the  3  groups  of  recruits  with  the  author  and  the  expert 
group.  The  results  are  shown  in  Table  8. 
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Table  8 


Correlations  of  Experts'  Cognitive  Structures  with  Novices 
(All  Experts  read  Original) 


Original 

Version 

Principled 

Revision 

Heuristic 

Revision 

Ehrhart 

.08 

.52* 

.61* 

Experts 
(n = 7) 

.10 

.55* 

.61* 

1 .  Ambassador 
Hillenbrandt 

.02 

.38* 

.44* 

2.  UGA  Military 

Historian  1 

.16 

.56* 

.55* 

3.  UGA  Military 

Historian  2 

.08 

.54* 

.59* 

4.  Major  Eunice 
(AFROTC 

Commandant 
and  Instructor) 

.11 

.51* 

.59* 

5.  Military  Employee 
in  Classified  Job 

.12 

.55* 

.66* 

6.  High  Level  Military 
Employee 

.03 

.41* 

.40* 

7.  Amateur  Military 
Historian 

*B  <  05 

.12 

.54* 

.62* 

As  you  can  see,  the  recruits  who  read  the  Principled  and  Heuristic  revisions  had  a  mental 
representation  closer  to  that  intended  by  the  author  than  did  those  who  read  the  Original  -  the  author's 
text.  That  is,  the  mental  representation  of  the  people  who  read  the  Principled  Revision  was  closer  to 
the  author's  own  mental  representation  than  the  representation  of  those  who  read  the  Original  was. 
The  results  with  the  rest  of  the  experts  were  the  same.  It  appears  that  our  Principled  Revision 
implemented  the  author's  intentions  better  than  the  author. 


10 


We  can  make  a  map  of  the  mental  structure  from  these  data  (Schvaneveldt,  1S89)  using 
Schvaneveldfs  Pathfinder  program.  Figure  1  shows  the  map  for  the  experts. 


Figure  1 


11 


The  map  makes  good  sense  If  you've  read  the  passage,  or  know  this  period  of  history.  The  passage  is 
centered  about  the  ‘graduated  response  strategy*  in  the  center;  arrayed  around  rt  are  the  various 
proposers  of  it  and  users  and  the  result  -  the  operatior  code-named  'Rolling  Thunder*  -  with  Rolling 
Thunder  linked  to  'failure'.  And  'success'  is  way  off  on  the  periphery  of  the  representation. 

But  those  who  read  the  Original  had  the  map  shown  in  Figure  2. 


Figure  2 


12 


They  think  Rolling  Thunder  was  a  success:  An  outcome  approximately  equivalent  to  reading  the  Bible 
and  concluding  that  Satan  was  the  Good  Guy. 

The  conclusion  is  that  Kintsch's  model  did  very  well  for  the  purposes  intended  here. 

The  Appendix  contains  a  full  report  on  these  two  studies. 

Partly  Accomplished  Objectives 

Theoretically  motivated  individual  differences  tests  of  the  abilities  needed  to  read  the  Original 
text,  according  to  Kintsch's  model,  have  been  developed.  These  include  tests  of  inference  making 
ability,  tests  of  the  ability  to  detect  that  there  is  a  need  for  an  inference  to  be  made,  and  tests  of  prior 
knowledge.  Criterion  tests  of  learning  from  the  text,  and  of  inference-making  in  the  text  have  also  been 
developed. 

In  two  visits  to  test  recruits,  pilot  tests  and  interviews  were  conducted  and  the  tests  have  been 
refined  and  improved.  Another  visit  is  planned  for  October  1990,  and  probably  the  rest  of  the 
proposed  visits  will  be  required  before  the  tests  are  ready.  This  is  mainly  because  the  tests  for  the 
construction-integration  ability  have  not  yet  been  successful.  This  test  is  based  on  a  connectionist 
paradigm.  A  visit  to  Kintsch's  laboratory  to  discuss  this  problem  is  planned  for  December,  1990.  It 
should  be  noted  that  development  of  the  construction-integration  tests  is  frankly  a  highly  speculative 
undertaking.  We  know  of  no  individual  differences  tests  of  connectionistic  abilities  (J.  McClelland, 
personal  communication,  August,  1989;  D.  Rumelhart,  personal  communication,  August  1989)  so 
unbroken  ground  is  being  plowed  here. 

Working  memory  tests  will  be  used  from  the  battery  developed  by  the  LAMP  program. 

Current  Status  of  Uncompleted  Research  Objectives 

The  main  uncompleted  research  objectives  are  (a)  to  relate  the  mental  representation  diagram 
produced  by  the  Kintsch  program  to  the  mental  representation  of  the  reader  (Specific  Aims  2a  +  b),  (b) 
to  develop  a  model  relating  the  individual  differences  to  learning  (Aim  5),  and  (c)  perform  the  empirical 
tests  (Aim  7).  These  empirical  tests  can  be  conducted  as  soon  as  the  rest  of  the  individual  differences 
tests  are  ready.  The  model  of  the  effects  of  individual  differences  on  learning  is  based  on  the  mediating 
and  moderating  effects  of  the  variables  measured  by  the  individual  differences  tests. 

The  other  uncompleted  objective  has  to  do  with  the  mental  representations  produced  by  the 
Kintsch  model.  Here  a  substantial  problem  has  arisen  along  with  an  opportunity,  in  that  the 
hierarchical  tree  representation  of  the  Miller  and  Kintsch  (1980)  model  appears  inadequate  for  many 
texts,  including  the  one  we  have  revised  Instead,  a  more  flexible  representation  is  required, 
corresponding  to  a  network.  A  network  is  more  general  and  flex,  .le  than  a  tree  because  any  tree  can 
be  represented  as  a  suitably  constrained  network,  and  a  network  can  also  represent  structures  other 
than  trees. 

Fortunately,  just  such  a  network  representation  is  now  available,  in  a  model  developed  by 
Althea  Turner.  She  worked  with  Kintsch  for  many  years,  and  her  model  is  descended  from  his.  Use  of 
her  model  was  foreshadowed  in  Specific  Aim  1  I  am  now  working  with  her  on  a  publishable  version  of 
(a)  the  model  and  (b)  an  experiment  in  which  her  model  accounted  for  about  30%  of  the  variance  in 
recall  of  about  400  individual  propositions  in  four  texts. 

Working  with  Thea,  I  am  applying  her  model  to  Air  Force  text  to  produce  a  network 
representation  of  it.  This  will  then  be  used  in  accomplishing  Specific  Aims  2a  and  b. 
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